Papers with disparity metrics
SAGED: A Holistic Bias-Benchmarking Pipeline for Language Models with Customisable Fairness Calibration (2025.coling-main)
Copied to clipboard
Xin Guan, Nate Demchak, Saloni Gupta, Ze Wang, Ediz Ertekin Jr., Adriano Koshiyama, Emre Kazim, Zekun Wu
| Challenge: | Existing benchmarks for large language models fail to detect bias due to limited scope, contamination, and lack of a fairness baseline. |
| Approach: | They propose a benchmarking pipeline to detect biases in large language models . they use metrics for max disparity, impact ratio, and bias concentration to analyze disparity . |
| Outcome: | SAGED(bias) is the first holistic benchmarking pipeline to address biases in large language models. |
TFDP: Token-Efficient Disparity Audits for Autoregressive LLMs via Single-Token Masked Evaluation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for auditing autoregressive Large Language Models for disparities are limited and expensive. |
| Approach: | They propose a method to detect disparities in autoregressive Large Language Models by token querying . they propose 'token-focused disparity probing' to measure disparities between sentence pairs . |
| Outcome: | The proposed method detects disparities with 42 times fewer output tokens than previous methods. |